{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7KBOS0yO_mi"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/search/question-answering/abstractive-question-answering.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/search/question-answering/abstractive-question-answering.ipynb)\n",
        "\n",
        "# Abstractive Question Answering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WL4goh8fPC5l"
      },
      "source": [
        "Abstractive question-answering focuses on the generation of multi-sentence answers to open-ended questions. It usually works by searching massive document stores for relevant information and then using this information to synthetically generate answers. This notebook demonstrates how Pinecone helps you build an abstractive question-answering system. We need three main components:\n",
        "\n",
        "- A vector index to store and run semantic search\n",
        "- A retriever model for embedding context passages\n",
        "- A generator model to generate answers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6uNNJmuPIVT"
      },
      "source": [
        "# Install Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZINfOdPyJHBZ",
        "outputId": "e1f0f6b3-9206-48d8-98f7-68ff2792f5be"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m486.2/486.2 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m179.1/179.1 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m11.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m22.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m60.0/60.0 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m300.0/300.0 kB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m63.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m30.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m66.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m57.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install -qU datasets pinecone-client==3.1.0 sentence-transformers torch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y0EU4eIbPTTL"
      },
      "source": [
        "# Load and Prepare Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "USvwSnsDvrdz"
      },
      "source": [
        "Our source data will be taken from the Wiki Snippets dataset, which contains over 17 million passages from Wikipedia. But, since indexing the entire dataset may take some time, we will only utilize 50,000 passages in this demo that include \"History\" in the \"section title\" column. If you want, you may utilize the complete dataset. Pinecone vector database can effortlessly manage millions of documents for you."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "689088f37f2a487fad3de9604ced8c90",
            "284ab50c8b8b4551863f8ed115ef82b6",
            "f183a892bcf44820808e5b465837189b",
            "ef60cea9781349ffa5cf792409ef8c84",
            "b987aac3f935492c82151e27c9e32660",
            "da1b483d8e094332b0e214eafe07db2d",
            "1b9b2d796e9d4fc09de7c3a83d506c9d",
            "53d880e05c7c48cb8b3dc92c19ffb7c8",
            "3379e63b1f46496ba22bb997c9823eff",
            "be1ac890a0d2458fb0e5e881b436bf63",
            "5a043cbb0af04f5f8cf61c7669233bbb"
          ]
        },
        "id": "bShG-f5IPPtG",
        "outputId": "f50b9514-b0ce-47f2-bf3e-90c56500cf35"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "689088f37f2a487fad3de9604ced8c90",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/4.58k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "# load the dataset from huggingface in streaming mode and shuffle it\n",
        "wiki_data = load_dataset(\n",
        "    'vblagoje/wikipedia_snippets_streamed',\n",
        "    split='train',\n",
        "    streaming=True\n",
        ").shuffle(seed=960)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MoPdaLFAvvci"
      },
      "source": [
        "We are loading the dataset in the streaming mode so that we don't have to wait for the whole dataset to download (which is over 9GB). Instead, we iteratively download records one at a time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VLjXCwKevy4K",
        "outputId": "b4918bc8-090e-4515-c8c6-eb0fd9b5405d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'wiki_id': 'Q7649565',\n",
              " 'start_paragraph': 20,\n",
              " 'start_character': 272,\n",
              " 'end_paragraph': 24,\n",
              " 'end_character': 380,\n",
              " 'article_title': 'Sustainable Agriculture Research and Education',\n",
              " 'section_title': \"2000s & Evaluation of the program's effectiveness\",\n",
              " 'passage_text': \"preserving the surrounding prairies. It ran until March 31, 2001.\\nIn 2008, SARE celebrated its 20th anniversary. To that date, the program had funded 3,700 projects and was operating with an annual budget of approximately $19 million. Evaluation of the program's effectiveness As of 2008, 64% of farmers who had received SARE grants stated that they had been able to earn increased profits as a result of the funding they received and utilization of sustainable agriculture methods. Additionally, 79% of grantees said that they had experienced a significant improvement in soil quality though the environmentally friendly, sustainable methods that they were\"}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# show the contents of a single document in the dataset\n",
        "next(iter(wiki_data))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "yHYNM6VXv1MJ"
      },
      "outputs": [],
      "source": [
        "# filter only documents with History as section_title\n",
        "history = wiki_data.filter(\n",
        "    lambda d: d['section_title'].startswith('History')\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_8sTBqS0v4RM"
      },
      "source": [
        "Let's iterate through the dataset and apply our filter to select the 50,000 historical passages. We will extract `article_title`, `section_title` and `passage_text` from each document."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "2453af0ab9f74f0c8af3d0a5b84cd1da",
            "c951a02c7e384328bf4b895e4ac23bc7",
            "5e4fb5220e9e462fbdd1d7f513b58602",
            "0e3cbf48197e4f50aafb28b0bad986ff",
            "7e6834870b064eecb5d292dd59aa1d48",
            "9c140d2e9e914a71a534929fa283e574",
            "9878b3e5c775499da23abe62904f59f3",
            "e3cb55cf46de4c7cb9d9fc82f2eb15da",
            "560c79f4011d45e98737717790fa4bcc",
            "3bedc4c4788242d8b316d5d690d34900",
            "742a7faaf32d4f5b8233b5f2c51128d2"
          ]
        },
        "id": "Ob0prowIzjJ9",
        "outputId": "246bf14f-8287-4058-b07e-a8bc198ac651"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2453af0ab9f74f0c8af3d0a5b84cd1da",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/50000 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from tqdm.auto import tqdm  # progress bar\n",
        "\n",
        "total_doc_count = 50000\n",
        "\n",
        "counter = 0\n",
        "docs = []\n",
        "# iterate through the dataset and apply our filter\n",
        "for d in tqdm(history, total=total_doc_count):\n",
        "    # extract the fields we need\n",
        "    doc = {\n",
        "        \"article_title\": d[\"article_title\"],\n",
        "        \"section_title\": d[\"section_title\"],\n",
        "        \"passage_text\": d[\"passage_text\"]\n",
        "    }\n",
        "    # add the dict containing fields we need to docs list\n",
        "    docs.append(doc)\n",
        "\n",
        "    # stop iteration once we reach 50k\n",
        "    if counter == total_doc_count:\n",
        "        break\n",
        "\n",
        "    # increase the counter on every iteration\n",
        "    counter += 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "RvTA40mq5FKj",
        "outputId": "c441b678-4d86-474e-8c2b-6249f1b83378"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-00c694a3-a2db-4d85-aa0f-9631d6ae8be6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>article_title</th>\n",
              "      <th>section_title</th>\n",
              "      <th>passage_text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Taupo District</td>\n",
              "      <td>History</td>\n",
              "      <td>was not until the 1950s that the region starte...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Sutarfeni</td>\n",
              "      <td>History &amp; Western asian analogues</td>\n",
              "      <td>Sutarfeni History strand-like pheni were Phena...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>The Bishop Wand Church of England School</td>\n",
              "      <td>History</td>\n",
              "      <td>The Bishop Wand Church of England School Histo...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Teufelsmoor</td>\n",
              "      <td>History &amp; Situation today</td>\n",
              "      <td>made to preserve the original landscape, altho...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Surface Hill Uniting Church</td>\n",
              "      <td>History</td>\n",
              "      <td>in perpetual reminder that work and worship go...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-00c694a3-a2db-4d85-aa0f-9631d6ae8be6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-45705f21-8b0d-4aa9-bbd0-c6cfdd7ecabd\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-45705f21-8b0d-4aa9-bbd0-c6cfdd7ecabd')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-45705f21-8b0d-4aa9-bbd0-c6cfdd7ecabd button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-00c694a3-a2db-4d85-aa0f-9631d6ae8be6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-00c694a3-a2db-4d85-aa0f-9631d6ae8be6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                              article_title  \\\n",
              "0                            Taupo District   \n",
              "1                                 Sutarfeni   \n",
              "2  The Bishop Wand Church of England School   \n",
              "3                               Teufelsmoor   \n",
              "4               Surface Hill Uniting Church   \n",
              "\n",
              "                       section_title  \\\n",
              "0                            History   \n",
              "1  History & Western asian analogues   \n",
              "2                            History   \n",
              "3          History & Situation today   \n",
              "4                            History   \n",
              "\n",
              "                                        passage_text  \n",
              "0  was not until the 1950s that the region starte...  \n",
              "1  Sutarfeni History strand-like pheni were Phena...  \n",
              "2  The Bishop Wand Church of England School Histo...  \n",
              "3  made to preserve the original landscape, altho...  \n",
              "4  in perpetual reminder that work and worship go...  "
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# create a pandas dataframe with the documents we extracted\n",
        "df = pd.DataFrame(docs)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lDYP3RkcPYdf"
      },
      "source": [
        "# Initialize Pinecone Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A5m5HHVu4War"
      },
      "source": [
        "The Pinecone index stores vector representations of our historical passages which we can retrieve later using another vector (query vector). To build our vector index, we must first establish a connection with Pinecone. For this, we need an API from Pinecone. You can get one for free from [here](https://app.pinecone.io/), and after that, we initialize the connection as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from pinecone import Pinecone\n",
        "\n",
        "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
        "api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
        "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
        "\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7CPJmTIy4XsL"
      },
      "source": [
        "Now we create a new index. We will name it \"abstractive-question-answering\" \u2014 you can name it anything we want. We specify the metric type as \"cosine\" and dimension as 768 because the retriever we use to generate context embeddings is optimized for cosine similarity and outputs 768-dimension vectors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "index_name = \"abstractive-question-answering\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "# check if index already exists (it shouldn't if this is first time)\n",
        "if index_name not in pc.list_indexes().names():\n",
        "    # if does not exist, create index\n",
        "    pc.create_index(\n",
        "        index_name,\n",
        "        dimension=768,\n",
        "        metric='cosine',\n",
        "        spec=spec\n",
        "    )\n",
        "    # wait for index to be initialized\n",
        "    while not pc.describe_index(index_name).status['ready']:\n",
        "        time.sleep(1)\n",
        "\n",
        "# connect to index\n",
        "index = pc.Index(index_name)\n",
        "# view index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHcCQVi4PcmE"
      },
      "source": [
        "# Initialize Retriever"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "udmF-qY24yQ1"
      },
      "source": [
        "Next, we need to initialize our retriever. The retriever will mainly do two things:\n",
        "\n",
        "- Generate embeddings for all historical passages (context vectors/embeddings)\n",
        "- Generate embeddings for our questions (query vector/embedding)\n",
        "\n",
        "The retriever will create embeddings such that the questions and passages that hold the answers to our queries are close to one another in the vector space. We will use a SentenceTransformer model based on Microsoft's MPNet as our retriever. This model performs quite well for comparing the similarity between queries and documents. We can use Cosine Similarity to compute the similarity between query and context vectors generated by this model (Pinecone automatically does this for us)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 552,
          "referenced_widgets": [
            "63a48723596443428d5feec8b725a7b3",
            "8a10bb67d84941cfa507442d4706aef4",
            "dec277846bda4b2291372c4ea97e286f",
            "076f43c0b00f4c2aac185bc87a1e3b84",
            "851e60de0349416199b79811a9f51f10",
            "b9506d2b30864b3f8345410cd66d16d8",
            "1bbc2b67d18a4e51a5038e29eaea9163",
            "c5ac5a175cbe440bad24da3149cc5830",
            "84e72bb9d09f4725a579681ff59e2f2a",
            "59f6a7f740514917bf66fb630ec1f894",
            "899626a5b381403aa931a48a06147867",
            "b9b5a0519f274ec38f9b7d6a80f56222",
            "acb26f43e3054456a217c06a1e1a137c",
            "83fa45faaaeb4d0980bdab856097c6db",
            "58cf29fbbab94a328519a4a7f0e35f36",
            "8be3f6b2c5a740c7a90959b1d2250a6c",
            "09fd4164894a4dba84ed89e6a408c01c",
            "d3c322b285f742c8aa37ce6ccf654b6d",
            "a90fc3a982e84001866ff3dfb386e18b",
            "f1366ec1d143421483848b62b95aba21",
            "13695cced79241d0b625a6674bc5b926",
            "20d24f01a8dd4d16b5c63725add5c367",
            "6e29fcdad32d475085d9b5f170c8385e",
            "0e5cf025869f402ba858bf804dc52df4",
            "2767aa86daa942889ee8865d745241cc",
            "3f38c70f07534346af96d6f293b6869c",
            "ebf26a8bfc524d4da35fe72824bb7d13",
            "cc8cf64efcc04f35b5a51a65629c5d1d",
            "d8cb92e041f84509b47fe5217f2e9abf",
            "daa58b71269e4ad4ab2413051e5f7c25",
            "28e468ffa31b469187432326b863516a",
            "5f6bdcf3339640318fe18f211457f836",
            "54bb2a4c300d4aad8c91eb5756be78a1",
            "943aeb633bb9488aaf478ff92d82a28c",
            "3650daf9110e4b0f8215bf3793064995",
            "975b257309ed445e816f715169099c8e",
            "77da1d35c3744d3687db58ac99717cd4",
            "8003888d39fe48498f201ca21268f6f6",
            "6632bf60bed641a38dcd3dea88b220a3",
            "cf422b0e35274f229fd1c01eee4eabd5",
            "6b2fa179e31241e880c1d4c4f9897961",
            "351c3c798af34e73b94e16a6998e06cf",
            "239f8d7e7585483384093371db15bb72",
            "da9bffda139a40eb933a85edfa187076",
            "b1de0e32ae3448b48b271c1d4ce379e8",
            "64d00090ea994356aec8ca2fa3950a1e",
            "d352d03a88094e5eb936ac736190942e",
            "9644b8475b4e4f91880713066e999675",
            "3f9f823757114e2498a9451ec5a74236",
            "f2642d28960c41fbaca0bc3e4eff8880",
            "0b5e45580cd74e478e5678b8aaca3968",
            "6403cd9ca9c14dc9bc6d5825de9d7950",
            "b9422c90ef9d469bb772a939000f8251",
            "7e923380122f4fbcae79ff41654ca120",
            "f28a61c429cc447fb6d55fc7cc5ea52f",
            "2ef601fda8ca46baa6472b8edfc9b7b7",
            "879be4c46d404012bc2fbd223ae2f517",
            "98594f58bd1e4c54b9617b12f4659d34",
            "7b060df1e1d94f35939fcd675e37ae84",
            "76248bb1c95a4536b6808889c8d67133",
            "a2fd9d835cea4046abc2b138ec1057d2",
            "de50517068fc4a569d67b2ae23459c3e",
            "5cc33cbaab6d450e8d5c7db406bcbb08",
            "9d03c179d8954fc4bdf990a22509d31e",
            "d1e76b46d4a4404d88a8921481521c9c",
            "921d1b5a40c1495280c54b6e43eafa77",
            "bf43dcff4d504d609b187f5d85dd3b4b",
            "9e5a1ee5e1984f7ba2dc420056b2f031",
            "67196173345447efba245cd24bbccf47",
            "b3b7678b02054bc2b08cc4e401efbd28",
            "7135122fbe7b479a8d05bdf0ba63f0ff",
            "3fe5113acb1542da9fe448b67bdf679e",
            "60665fb7beb74c678526686f35d03bba",
            "586050cac17b4b0fb477e02334a70a9d",
            "b9091d48252b4932ba7934acad3745e3",
            "d1493961ff1743d3adee561ce2d11e0d",
            "8e265e6420af4c708fbc938ecd1f0270",
            "e0d9205076fd4c41b1793aa263c0f822",
            "451325f7fcb548219e0ba1ef7b3eb41b",
            "5a727aee6d8148c79876d8d43434dec8",
            "a1b913c9da654d7f8854a7130417f7ed",
            "3a4e8d9336bb4d539f5cc8b229be1d2d",
            "75362289a1c148c88bbe6616fba1bfb4",
            "c255044a41194deebf74c54f90bc6621",
            "0ec0fd443d7d4a29b16bd37355ce419a",
            "de15a07bc8e54107a5d1ddff90639aa8",
            "73e99b832ad24762ac5f7764c4be3d03",
            "f992ceb9d9f347eaac52d5a08f63f48b",
            "51aef90927514d60b95b560d8a14d231",
            "5827e442de444c4d9900fd997b8716fc",
            "bff2f6b0b3624443b29f4ae2cdf82b52",
            "e73efe32b9404d01aba66dfb550a7e5c",
            "d8a67c6cea8843d3b9e51546dab0a1a5",
            "9971f06352a74bda8514f104681fb46e",
            "9f38cf2ee15849c0b8bc3e926a4d63c6",
            "7343a3d4cb7c426585c07de15406e0c2",
            "414bb083d7274422bc9da8f94758bfdc",
            "a73fa0b895d2429185a5bdd8d16350da",
            "2a6fac55d6d549febe91b3c94134c6ab",
            "32b505a42a92449b980ee9b2bc77f5fe",
            "18e4a1d7009049558e12533dd053972c",
            "5319f32588764a0d8eb14dabf5584009",
            "8418ad29994a43209abf3a4c42971a81",
            "3f87b0be49e543f8b85f575d5b7baa54",
            "f6868c87a6d547b5b287b9ed164b922c",
            "942ec90573614d4f8d012dd630e53f75",
            "b0ce3d8bcf65464480660de32dfdd7a1",
            "cb44d34d5bcf46d785b5f8168124bc1d",
            "1ecb36e997904d768bc8d28c6c2753a5",
            "3e5ce752c531428d8122909edc50b01c",
            "ac5ad08d04c74acab5fc6ffa4d1a9e83",
            "fe81ff7a202b4b0ca79e8e7417e88815",
            "9af1a1f161434c11b4526d49f84360e1",
            "a2a493d5e11b4f50a1f35a44357c825e",
            "38dd55fefacb43beb6d6197d87474593",
            "af25fe16426e44a0b2d99fda1b27bf73",
            "8d269802b10a4e06a9657942f567709d",
            "6da115ccb21044e486d8188db0424c75",
            "e362c30732604587adc3327fc57a8174",
            "8be748121de8409e8b8a32ae9b6668c8",
            "358b5bdba5874ded990950b905047588",
            "540f5b917d554e03be27de7779ed18d6",
            "54b5218528fb4ffab5bc54e6a3e7f171",
            "9f07d2ce61c942b182bee4d2d5682cf6",
            "1be9705ef3fb4c6383745d7f38aa594d",
            "0dde3152edf84b3e898bef35619f60c7",
            "73aa9f8665534c90ac43753369e9e2ab",
            "3019a5e71dae42cebe9a72149950dda6",
            "bc7b74df024d442aa6f5cf2e2897b029",
            "a31c933b73164719980c81773350a481",
            "801d5542572a477d85b9e6f15e7a9095",
            "f05ee268957941d186f2ac029a683e57",
            "43fccf037a7b485bba1f6a8959fc148f",
            "2a0ab3fd886942019342881a185fc125",
            "b103111a8c4d4f9897fda3e2386ec190",
            "ac1bc20313fa4beda6748485fba0b984",
            "39b8afb2838241fd875dee0259c0b5f5",
            "147e188592c949dbbd85b1e2406621b9",
            "985420170fad4b009aa298de7432b63a",
            "c04f4dae1604457bb96d85f1e0243e14",
            "c650a28ca0534488bf88f7c42785d774",
            "404eebec97f64e77a828e73119ace05f",
            "a268707103a34bef9ae725f1953d2e0b",
            "d3d53aa382f146d08451ad5dfa33a01f",
            "fc65a3dd0afd4268b794a2aa2d82c3d9",
            "61870ec539a64e53bb97243a1b02ebf9",
            "f13586fe86834f028596f07dc6259d06",
            "ad860d337fd84da9afe84ddce20b753a",
            "8131ed8a6a8e4f98956f8cf03ab379ec",
            "68c8b79f963e4f93a403a4cbba91cbc9",
            "3be0d48e44154d97a0f04fc082ea440d",
            "47680d0d73dd4a78a64adcb549a3f3eb",
            "fd60d6db9b094d51b935b57871b1ef88",
            "938ac82f73a24f49b3bb88c5bdfc22ff"
          ]
        },
        "id": "45dRrEZNPdXu",
        "outputId": "5ac1e06e-8423-4083-9244-3b344ea3d073"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "63a48723596443428d5feec8b725a7b3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)e933c/.gitattributes:   0%|          | 0.00/737 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b9b5a0519f274ec38f9b7d6a80f56222",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6e29fcdad32d475085d9b5f170c8385e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cbe6ee933c/README.md:   0%|          | 0.00/9.85k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "943aeb633bb9488aaf478ff92d82a28c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)e6ee933c/config.json:   0%|          | 0.00/591 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b1de0e32ae3448b48b271c1d4ce379e8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)ce_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2ef601fda8ca46baa6472b8edfc9b7b7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)33c/data_config.json:   0%|          | 0.00/15.7k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bf43dcff4d504d609b187f5d85dd3b4b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading pytorch_model.bin:   0%|          | 0.00/438M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e0d9205076fd4c41b1793aa263c0f822",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)nce_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "51aef90927514d60b95b560d8a14d231",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/239 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "32b505a42a92449b980ee9b2bc77f5fe",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)e933c/tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ac5ad08d04c74acab5fc6ffa4d1a9e83",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/383 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "540f5b917d554e03be27de7779ed18d6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)933c/train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "43fccf037a7b485bba1f6a8959fc148f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cbe6ee933c/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d3d53aa382f146d08451ad5dfa33a01f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)6ee933c/modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "SentenceTransformer(\n",
              "  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: MPNetModel \n",
              "  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})\n",
              "  (2): Normalize()\n",
              ")"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import torch\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "# set device to GPU if available\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "# load the retriever model from huggingface model hub\n",
        "retriever = SentenceTransformer(\"flax-sentence-embeddings/all_datasets_v3_mpnet-base\", device=device)\n",
        "retriever"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jo5MN_GKPeDy"
      },
      "source": [
        "# Generate Embeddings and Upsert"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hj0NdB-kU9Yt"
      },
      "source": [
        "Next, we need to generate embeddings for the context passages. We will do this in batches to help us more quickly generate embeddings and upload them to the Pinecone index. When passing the documents to Pinecone, we need an id (a unique value), context embedding, and metadata for each document representing context passages in the dataset. The metadata is a dictionary containing data relevant to our embeddings, such as the article title, section title, passage text, etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 118,
          "referenced_widgets": [
            "dccce334ef2e4c6ea545adb9a5d6f3f7",
            "f5c21fd3f57449809967e882733da821",
            "ab75e9902ea24af3b27ddbf67377ba56",
            "1e2620bdbb614e3ca4b46d36485fcd4d",
            "5b46568ad14f4a698cf87c015c738600",
            "d15729fecb57457b98db11245a5c1b47",
            "d4301e89b40043e38aa1214b424141fa",
            "a5fadab8bf8a4d6cb679e5b1053a057d",
            "5ddb43098ba5403893b718d41ef0f8a7",
            "f9599632cc264a2290bb7d0cbedfdf78",
            "89b24c20d2e24c468c4c39f93d698ab9"
          ]
        },
        "id": "4OqB7bBePia-",
        "outputId": "0a7475b2-b731-4681-8ae4-a4151442e291"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "dccce334ef2e4c6ea545adb9a5d6f3f7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/782 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'dimension': 768,\n",
              " 'index_fullness': 0.1,\n",
              " 'namespaces': {'': {'vector_count': 50001}},\n",
              " 'total_vector_count': 50001}"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# we will use batches of 64\n",
        "batch_size = 64\n",
        "\n",
        "for i in tqdm(range(0, len(df), batch_size)):\n",
        "    # find end of batch\n",
        "    i_end = min(i+batch_size, len(df))\n",
        "    # extract batch\n",
        "    batch = df.iloc[i:i_end]\n",
        "    # generate embeddings for batch\n",
        "    emb = retriever.encode(batch[\"passage_text\"].tolist()).tolist()\n",
        "    # get metadata\n",
        "    meta = batch.to_dict(orient=\"records\")\n",
        "    # create unique IDs\n",
        "    ids = [f\"{idx}\" for idx in range(i, i_end)]\n",
        "    # add all to upsert list\n",
        "    to_upsert = list(zip(ids, emb, meta))\n",
        "    # upsert/insert these records to pinecone\n",
        "    _ = index.upsert(vectors=to_upsert)\n",
        "\n",
        "# check that we have all vectors in index\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WFKPJ0WtPmAw"
      },
      "source": [
        "# Initialize Generator"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RraVCG_hcv3"
      },
      "source": [
        "We will use ELI5 BART for the generator which is a Sequence-To-Sequence model trained using the \u2018Explain Like I\u2019m 5\u2019 (ELI5) dataset. Sequence-To-Sequence models can take a text sequence as input and produce a different text sequence as output.\n",
        "\n",
        "The input to the ELI5 BART model is a single string which is a concatenation of the query and the relevant documents providing the context for the answer. The documents are separated by a special token &lt;P>, so the input string will look as follows:\n",
        "\n",
        ">question: What is a sonic boom? context: &lt;P> A sonic boom is a sound associated with shock waves created when an object travels through the air faster than the speed of sound. &lt;P> Sonic booms generate enormous amounts of sound energy, sounding similar to an explosion or a thunderclap to the human ear. &lt;P> Sonic booms due to large supersonic aircraft can be particularly loud and startling, tend to awaken people, and may cause minor damage to some structures. This led to prohibition of routine supersonic flight overland.\n",
        "\n",
        "More detail on how the ELI5 dataset was built is available [here](https://arxiv.org/abs/1907.09190) and how ELI5 BART model was trained is available [here](https://yjernite.github.io/lfqa.html).\n",
        "\n",
        "Let's initialize the BART model using transformers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177,
          "referenced_widgets": [
            "ef03146195dd47e99dacc9e174235e79",
            "126dc23fdbf04b9ebe3d8ed403982f9c",
            "b70c5bb1d36145cb9c3b032ea851e26c",
            "d573a3166a964f87a2e6362678245aff",
            "3cea067f728f4d0e8c4397af021a5636",
            "ecbbb44bdb774dbfb79b92a0b4a0e744",
            "1c590b987bd049ea8eefa17d511be830",
            "722169eb54db401cbdb11a249a8de7d6",
            "b71c6b0549794e5a875782116d1802a8",
            "e0d61c9e04eb42198fe73b97e3300ca5",
            "97135f3cd6bc4d2bb47f60ff95d4b11c",
            "e190f8b530e3452a8ca63446b525737d",
            "a44cd7e0a66e4e9e8cb6310a1346d838",
            "7a7563d8b5fc4171ab4afa163a460642",
            "465fe0a6fa174bb284bee9351996d162",
            "ca1286424a8746f0a605bf252152cf37",
            "c362b0bffe8a4f0f8fe09d0aa117fda3",
            "5c1b87355b1b4ced8f542541cc2a0e24",
            "47fc92e73b0d4455a181d03b8b225e25",
            "b80a56a40e754c4e90fe6249d4ab6472",
            "098778c25b2447e39e2602cf7fa4bf60",
            "6c0f5c1c64e14d8290aeb280e9acbeb6",
            "66aab4448c594f6aad1be2237a8318c9",
            "0088732428a2470db9656c30a4d24cfa",
            "2ff32570493949899409490e8eba8d6d",
            "f5b7c5c27c5d475b889d3040daf36547",
            "fe54a1eafeea4b87bfa8858bb677d1d3",
            "418e29fbf9fa4a649035d6cf625e243c",
            "2b44c5cec7c54ce4a2792248456094aa",
            "144c63a2c68e497f9f2c5930d9792455",
            "507b555a420c451295213023767477dd",
            "a07961a05bfb4460b61a827b0696fdbc",
            "d5c9c9e52c444e8f9ae8720f732fc590",
            "e83a1ef44ef444d8aa8bcb99d17ee2b3",
            "042e405cca93460abd71fe1c7a03a674",
            "dc2cc3500c844e33beafa0f33d3dd74b",
            "3d5cdd6933224930a9b54aedea2f90ff",
            "275c7a3db6c343318e5f80c5afc7787b",
            "bbbe4208624645ba8b8110fc2c22df90",
            "40ece01d305a47ad9a85deb1f9819075",
            "3103f676e5c24c27ac1f1eb7abd1c7b7",
            "1df57d119d244056a538b64a194fe001",
            "4e3ca3b942ad47ed8b2d29fcf2d44bdd",
            "32154720a3694013a37e2505071ade64",
            "0181dc586de148e3821a8929ee9c096d",
            "6a8a3c4c8f7445e39f3e20cd02f96b3d",
            "b5f6c0ca6d9e4d2b848201b448c4caeb",
            "0a5adad325ff43189d411ba082435265",
            "f7e0396252134e3e9734b09dc22c2d19",
            "ed1cea54f7a94f29a9075f31fad8a1f3",
            "3002014c1b5e421d9c73f894ee231205",
            "4d7b6894a1364d078d53d45fc6e8242a",
            "3a29c782a8574825946342f3d861b993",
            "1fb4285cf1ed4817a832d73c938eb530",
            "608f99e0db1a4b27b16215d38d4e6d1a"
          ]
        },
        "id": "76QyADZ0Pqki",
        "outputId": "4cbf90e7-dcf5-4e51-eaf9-e8e5ffd69379"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ef03146195dd47e99dacc9e174235e79",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)olve/main/vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e190f8b530e3452a8ca63446b525737d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "66aab4448c594f6aad1be2237a8318c9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/27.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e83a1ef44ef444d8aa8bcb99d17ee2b3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)lve/main/config.json:   0%|          | 0.00/1.32k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0181dc586de148e3821a8929ee9c096d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading pytorch_model.bin:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from transformers import BartTokenizer, BartForConditionalGeneration\n",
        "\n",
        "# load bart tokenizer and model from huggingface\n",
        "tokenizer = BartTokenizer.from_pretrained('vblagoje/bart_lfqa')\n",
        "generator = BartForConditionalGeneration.from_pretrained('vblagoje/bart_lfqa').to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RBr1GqvwkI3W"
      },
      "source": [
        "All the components of our abstract QA system are complete and ready to be queried. But first, let's write some helper functions to retrieve context passages from Pinecone index and to format the query in the way the generator expects the input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "mJfIe7CtlsWo"
      },
      "outputs": [],
      "source": [
        "def query_pinecone(query, top_k):\n",
        "    # generate embeddings for the query\n",
        "    xq = retriever.encode([query]).tolist()\n",
        "    # search pinecone index for context passage with the answer\n",
        "    xc = index.query(vector=xq, top_k=top_k, include_metadata=True)\n",
        "    return xc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "wiOZ2yWDHI8p"
      },
      "outputs": [],
      "source": [
        "def format_query(query, context):\n",
        "    # extract passage_text from Pinecone search result and add the <P> tag\n",
        "    context = [f\"<P> {m['metadata']['passage_text']}\" for m in context]\n",
        "    # concatinate all context passages\n",
        "    context = \" \".join(context)\n",
        "    # contcatinate the query and context passages\n",
        "    query = f\"question: {query} context: {context}\"\n",
        "    return query"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yA0f1Xyxq-Fp"
      },
      "source": [
        "Let's test the helper functions. We will query the Pinecone index function we created earlier with the `query_pinecone` to get context passages and pass them to the `format_query` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NSm2nj11mR87",
        "outputId": "21fa653e-17dc-4fdb-c862-20e8d7b3f8b8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'matches': [{'id': '3593',\n",
              "              'metadata': {'article_title': 'Electric power system',\n",
              "                           'passage_text': 'Electric power system History In '\n",
              "                                           '1881, two electricians built the '\n",
              "                                           \"world's first power system at \"\n",
              "                                           'Godalming in England. It was '\n",
              "                                           'powered by two waterwheels and '\n",
              "                                           'produced an alternating current '\n",
              "                                           'that in turn supplied seven '\n",
              "                                           'Siemens arc lamps at 250 volts and '\n",
              "                                           '34 incandescent lamps at 40 volts. '\n",
              "                                           'However, supply to the lamps was '\n",
              "                                           'intermittent and in 1882 Thomas '\n",
              "                                           'Edison and his company, The Edison '\n",
              "                                           'Electric Light Company, developed '\n",
              "                                           'the first steam-powered electric '\n",
              "                                           'power station on Pearl Street in '\n",
              "                                           'New York City. The Pearl Street '\n",
              "                                           'Station initially powered around '\n",
              "                                           '3,000 lamps for 59 customers. The '\n",
              "                                           'power station generated direct '\n",
              "                                           'current and',\n",
              "                           'section_title': 'History'},\n",
              "              'score': 0.69118017,\n",
              "              'values': []}],\n",
              " 'namespace': ''}"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = \"when was the first electric power system built?\"\n",
        "result = query_pinecone(query, top_k=1)\n",
        "result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "6yoylQFO4joX"
      },
      "outputs": [],
      "source": [
        "from pprint import pprint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tPxVoo6am8ls",
        "outputId": "aa7311cb-4b0f-4988-91fb-05962605f1c4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('question: when was the first electric power system built? context: <P> '\n",
            " \"Electric power system History In 1881, two electricians built the world's \"\n",
            " 'first power system at Godalming in England. It was powered by two '\n",
            " 'waterwheels and produced an alternating current that in turn supplied seven '\n",
            " 'Siemens arc lamps at 250 volts and 34 incandescent lamps at 40 volts. '\n",
            " 'However, supply to the lamps was intermittent and in 1882 Thomas Edison and '\n",
            " 'his company, The Edison Electric Light Company, developed the first '\n",
            " 'steam-powered electric power station on Pearl Street in New York City. The '\n",
            " 'Pearl Street Station initially powered around 3,000 lamps for 59 customers. '\n",
            " 'The power station generated direct current and')\n"
          ]
        }
      ],
      "source": [
        "# format the query in the form generator expects the input\n",
        "query = format_query(query, result[\"matches\"])\n",
        "pprint(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OY8LSqk7rdPS"
      },
      "source": [
        "The output looks great. Now let's write a function to generate answers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "2fxIoNqeoEPD"
      },
      "outputs": [],
      "source": [
        "def generate_answer(query):\n",
        "    # tokenize the query to get input_ids\n",
        "    inputs = tokenizer([query], max_length=1024, return_tensors=\"pt\").to(device)\n",
        "    # use generator to predict output ids\n",
        "    ids = generator.generate(inputs[\"input_ids\"], num_beams=2, min_length=20, max_length=40)\n",
        "    # use tokenizer to decode the output ids\n",
        "    answer = tokenizer.batch_decode(ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n",
        "    return pprint(answer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cYXFSBqSsZMx",
        "outputId": "2a395994-31c4-463d-c18d-a18c427eeae6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('The first electric power system was built in 1881 at Godalming in England. '\n",
            " 'It was powered by two waterwheels and produced alternating current that in '\n",
            " 'turn supplied seven Siemens arc lamps')\n"
          ]
        }
      ],
      "source": [
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gYWACzIotWQY"
      },
      "source": [
        "As we can see, the generator used the provided context to answer our question. Let's run some more queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6iFLxPPvx2Tm",
        "outputId": "48c3ceb2-d1e1-4e36-87a1-7d8df5164b17"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('The first wireless message was sent in 1866 by Mahlon Loomis, who had a kite '\n",
            " 'on a mountaintop 14 miles apart. The kite was connected to a cable')\n"
          ]
        }
      ],
      "source": [
        "query = \"How was the first wireless message sent?\"\n",
        "context = query_pinecone(query, top_k=5)\n",
        "query = format_query(query, context[\"matches\"])\n",
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI2fTt_A5Ozf"
      },
      "source": [
        "To confirm that this answer is correct, we can check the contexts used to generate the answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "15WZSeZw3mNY",
        "outputId": "b0813d34-933b-4d38-d200-5df252527c83"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "by electrostatic induction or electromagnetic induction, which had too short a range to be practical. In 1866 Mahlon Loomis claimed to have transmitted an electrical signal through the atmosphere between two 600 foot wires held aloft by kites on mountaintops 14 miles apart. Thomas Edison had come close to discovering radio in 1875; he had generated and detected radio waves which he called \"etheric currents\" experimenting with high-voltage spark circuits, but due to lack of time did not pursue the matter. David Edward Hughes in 1879 had also stumbled on radio wave transmission which he received with his carbon microphone\n",
            "---\n",
            "the east coast of India, then on to Penang, Malacca, Singapore, Batavia (current Jakarta), to finally reach Darwin, Australia. It was the first direct link between Australia and Great Britain. The company that laid the first part of the cable took the name of Falmouth, Gibraltar and Malta Telegraph Company and had been founded in 1869. This company later operated as the Eastern Telegraph Company from Mount Pleasant in Gibraltar and eventually became Cable & Wireless.\n",
            "The first telephones were introduced to Gibraltar in 1886 by a private company which was later taken over by the colonial authorities. The first wireless\n",
            "---\n",
            "audio distance records, and were heard as far west as Hawaii. They were also received in Paris, France, which marked the first transmission of speech across the Atlantic.\n",
            "With the entrance of the United States into World War I in April 1917 the federal government took over full control of the radio industry, and it became illegal for civilians to possess an operational radio receiver. However NAA continued to operate during the conflict. In addition to time signals and weather reports, it also broadcast news summaries received by troops on land and aboard ships in the Atlantic. Effective April 15, 1919\n",
            "---\n",
            "Message from space (science fiction) For other uses, see Message from Space (disambiguation).\n",
            "\"Message from space\" is a type of \"first contact\" theme in science fiction . Stories of this type involve receiving an interstellar message which reveals the existence of other intelligent life in the universe. History An early short story, A Message from Space (Joseph Schlossel, Weird Tales, March 1926) tells of an amateur who builds a ham TV set and suddenly sees an alien, The latter one realises it is being watched and tells its soap opera story. The verdict of Everett Franklin Bleiler: \"original ideas, but clumsy\n",
            "---\n",
            "radio operators, interested in a practical benefit from their hobby, and jewelers, who previously had been reliant on time services transmitted over telegraph wires, which had a reputation for being both expensive and of questionable reliability, especially compared to the free and very accurate NAA transmissions.\n",
            "NAA's original transmitters were only capable of producing the dots-and-dashes of Morse code. The later development of vacuum tube transmitters made audio transmissions practical, and in 1915 the American Telephone and Telegraph Company (AT&T) received permission from the Navy to conduct a series of tests at the NAA facility. These experimental transmissions set impressive new\n",
            "---\n"
          ]
        }
      ],
      "source": [
        "for doc in context[\"matches\"]:\n",
        "    print(doc[\"metadata\"][\"passage_text\"], end='\\n---\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bH-K7WHR9z8m"
      },
      "source": [
        "In this case, the answer looks correct. If we ask a question and no relevant contexts are retrieved, the generator will typically return nonsensical or false answers, like with this question about COVID-19:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T1naD6pRlmWd",
        "outputId": "996f5c5a-5809-4ab7-9ccf-c932e6d0c225"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('COVID-19 is a zoonotic disease, which means that it is a virus that is '\n",
            " 'transmitted from one animal to another. It is not a virus that can be '\n",
            " 'transmitted from person')\n"
          ]
        }
      ],
      "source": [
        "query = \"where did COVID-19 originate?\"\n",
        "context = query_pinecone(query, top_k=3)\n",
        "query = format_query(query, context[\"matches\"])\n",
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W3gX5xQY-jKJ",
        "outputId": "8c5fdefc-cb6f-4c69-af83-94b44b03371f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "to establish with certainty which diseases jumped from other animals to humans, but there is increasing evidence from DNA and RNA sequencing, that measles, smallpox, influenza, HIV, and diphtheria came to humans this way. Various forms of the common cold and tuberculosis also are adaptations of strains originating in other species.\n",
            "Zoonoses are of interest because they are often previously unrecognized diseases or have increased virulence in populations lacking immunity. The West Nile virus appeared in the United States in 1999 in the New York City area, and moved through the country in the summer of 2002, causing much distress. Bubonic\n",
            "---\n",
            "plague is a zoonotic disease, as are salmonellosis, Rocky Mountain spotted fever, and Lyme disease.\n",
            "A major factor contributing to the appearance of new zoonotic pathogens in human populations is increased contact between humans and wildlife. This can be caused either by encroachment of human activity into wilderness areas or by movement of wild animals into areas of human activity. An example of this is the outbreak of Nipah virus in peninsular Malaysia in 1999, when intensive pig farming began on the habitat of infected fruit bats. Unidentified infection of the pigs amplified the force of infection, eventually transmitting the virus\n",
            "---\n",
            "man killed and twenty-nine died of disease.\n",
            "---\n"
          ]
        }
      ],
      "source": [
        "for doc in context[\"matches\"]:\n",
        "    print(doc[\"metadata\"][\"passage_text\"], end='\\n---\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCUqB_MrAABI"
      },
      "source": [
        "Let\u2019s finish with a final few questions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F_oWypShW_rk",
        "outputId": "3386df79-46b3-4fd2-9912-cbc2b1375e5c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('The War of Currents was a series of events in the early 1900s between Edison '\n",
            " 'and Westinghouse. The two companies were competing for the market share of '\n",
            " 'electric power in the United States')\n"
          ]
        }
      ],
      "source": [
        "query = \"what was the war of currents?\"\n",
        "context = query_pinecone(query, top_k=5)\n",
        "query = format_query(query, context[\"matches\"])\n",
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bJ1-OsmT0duI",
        "outputId": "51613a75-6727-4e4b-ea68-f10987467779"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('The first person to walk on the moon was Neil Armstrong in 1969. He walked '\n",
            " 'on the moon in 1969. He was the first person to walk on the moon.')\n"
          ]
        }
      ],
      "source": [
        "query = \"who was the first person on the moon?\"\n",
        "context = query_pinecone(query, top_k=10)\n",
        "query = format_query(query, context[\"matches\"])\n",
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JU6DH-ah1g1t",
        "outputId": "c335e0ee-e248-46ec-955c-5fd07c363d99"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('The Space Shuttle was the most expensive project in the history of NASA. It '\n",
            " 'cost about $10 billion to build.')\n"
          ]
        }
      ],
      "source": [
        "query = \"what was NASAs most expensive project?\"\n",
        "context = query_pinecone(query, top_k=3)\n",
        "query = format_query(query, context[\"matches\"])\n",
        "generate_answer(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-cQPSkG9cFY"
      },
      "source": [
        "As we can see, the model can generate some decent answers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W1dAlvpeJH7q"
      },
      "source": [
        "# Example Application"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZLxMUC8JFnO"
      },
      "source": [
        "To try out an application like this one, see this [example application](https://huggingface.co/spaces/pinecone/abstractive-question-answering)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}